@alexsu52 Requesting review as per @MaximProshin 's guideline
Unfortunately, I can't assign the PR myself because I have no necessary rights

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 29.95%. Comparing base (9c00000) to head (e5db8cc).
Report is 1 commits behind head on develop.

@@             Coverage Diff              @@
##           develop    #2636       +/-   ##
============================================
- Coverage    91.21%   29.95%   -61.26%     
============================================
  Files          494      494               
  Lines        45775    45775               
============================================
- Hits         41753    13713    -28040     
- Misses        4022    32062    +28040     

@alexsu52 I think I fixed the code, could you please approve the workflow?

Please check that you have pushed all the changes because I don't see the changes you mentioned in the description.

Good evening. Sorry, those initial changes are irrelevant. I changed the code a bit because the initial code was not passing the pipeline.

Please, provide local results of the test run.

Will send screenshots in a bit

@alexsu52

Command

pytest tests/post_training/test_quantize_conformance.py::test_weight_compression -s --data=tests/post_training/data/ -k tinyllama_int8_data_free_backend_PT

Output

@alexsu52

Command

pytest tests/post_training/test_quantize_conformance.py::test_weight_compression -s --data=tests/post_training/data/ -k tinyllama_int8_data_free_backend_PT

Output

Your command does not run any test. There are 0 selected tests in your screenshot. I run your test using the following command:

 pytest tests/post_training/test_quantize_conformance.py::test_weight_compression -s --data=tests/post_training/data/ -k tinyllama_int8_data_free_backend_TORCH

and got the following error:

@alexsu52 Oh, I just used the command provided in the issue, I'll fix it and tag you asap. Thanks for the feedback

@alexsu52 Hi, could you verify the logic I followed? Looks good so far. You can check the updated PR description for a heads-up

We have the similar test pipeline for PyTorch model in PTQ test:

nncf/tests/post_training/test_quantize_conformance.py

Line 199 in fa1a4ce

def test_ptq_quantization(

You need to reproduce the same pipeline for weight compression test.

Will do

@alexsu52 Good morning, any updates? Do you need me to help you with anything?

@alexsu52 Good morning, any updates? Do you need me to help you with anything?

Hi, I tried to run your PR and got a runtime error. I'll come back with comments after I've done some experiments.

@alexsu52 Good morning, any updates? Do you need me to help you with anything?

Hi, I tried to run your PR and got a runtime error. I'll come back with comments after I've done some experiments.

Oh that's weird. I'll try to run it as well

@alexsu52 I removed the problematic line of code, I must have returned it by accidentally undo-ing right before the commit

@alexsu52 did it run?

@alexsu52 did it run?

Thanks for your update. Please pay attention to my comments.

@alexsu52 did it run?

Thanks for your update. Please pay attention to my comments.

Good day! Yup, thanks for the review

As I understand, based on your implementation, you are trying to implement the following test flow:

1. Create FP32 PyTorch HF model
2. Export FP32 PyTorch HF model to FP32 OpenVINO model and save to fp32_model_dir
3. Compress FP32 PyTorch HF model to INT8
4. Export INT8 PyTorch HF model to INT8 OpenVINO model and save to output_model_dir
5. Calculate the number of int8 and int4 operations by INT8 OpenVINO model.
6. Check the number of in8 and int4 operations with references.
7. Calculate the similarity metric between FP32 OpenVINO model and INT8 OpenVINO model. The similarity metric is calculated between OpenVINO models for inference optimization on CPU.
8. Check the similarity metric with reference.

Is this correct statement?

Yes, except in the Step 4. I didn't export it as OpenVINO model

@alexsu52 Thanks for a detailed review, I implemented all the changes and cleaned up the remaining code, could you review it one last time?

@alexsu52 FYI, here's the output:

@alexsu52 FYI, here's the output:

I have run internal build for validation your changes. It takes some time.

@alexsu52 FYI, here's the output:

I have run internal build for validation your changes. It takes some time.

Yup I see it, should be done in a few minutes

UPD: @alexsu52 Pipeline Passed

Thanks for guidance, have a great day